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In an effort to meet the need for a more 
comprehensive program evaluation, staff at Moraine Valley Community 
College developed* a questionnaire and interview schedule to obtain 
feedback from employers concerning career programs. Questionnaires 
were sent to all employers who hired graduates of occupational 
0 programs between 1969-71; over 80% returned the questionnaires. A 

- • sample were selected for the interview phase as well; interview 

• schedules were completed for 60S of the sample. .Factor analysis of 

^feV the original 25-item questionnaire designed to evaluate technical, 

^Vc r human relations and problem-solving skills revealed that employers 

: v evaluated graduates using a two-dimensional structure that included 

only technical and human relations skills* Results showed that: (1) 
1;^ employers rated as effective or highly effective their employees • 

fV training; (2) evaluations for public service, health and business 

^ programs were similar; (3) public service graduates were rated 

: 'WS~ slightly higher in technical skills than business and health 

graduates; (4) personality factors were mentioned most Qften as areas 
: "I" . of strength or weakness; (5) the chances for advancement of half the 

| r graduates were rated higher than the chances of .other emplpyees in 

fttf similar positions; (6) supervisors were interested in hiring 

|g graduates; and (7) a positive correlation exists between college 

§r evaluation and job performance evaluation. Program changes were 

f\_ recommended on the basis of these data. (KM) 
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Lynn Willett and William E. Piland 
Moraine Valley Community College 
\XJ Palos Hills, Illinois 

Effective community college occupational program planning requires a 
comprehensive evaluation system. Typical components of existing evaluation 
systems focus on student enrollment, attrition, number of graduates, unit 
costs and cost-benefit ratios. Additionally, most authors (Henderson, 1970; 
Butler and York, 1971; Suffolk County, 1969; and Banathy, 1973). advocate the 
use of follow-up surveys of the graduates and their employers. 

Little evidence existed, however, that community colleges had conducted 
employer follow-up surveys. For example, in a national survey of technical 
and community colleges, American College Testing Corporation (Garland and ■ 
Carmondy, 1970) found only one two-year college (Brandywine College, 1970) 
which reported conducting an employer survey. Brandywine f s survey was 

limited to two open-ended questions. Generally, community colleges which 

p?_- 

fe: conduct follow-up study activities survey only the graduates. 

In an effort to meet the need for a more comprehensive program evalua- 
tion scheme, staff at Moraine Valley Community College (MVCC) have identified 
career program evaluation procedures which include feedback from employers 
(see Figure 1). The following activities comprise the "VCC employer eval- 

CO 

uation system: identification of job performance competencies, designation 
of program opals, development of behavioral objectives, identifying major 
skill areas, instrument development and revision, and data reporting. 



* Paper presented at the North Central Special Interest Group for Conmunity- 
Junior College Research-American Educational Research Association, Ann Arbor, 
Michigan, July 13, 1973. 
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Procedures 

The college's Follow-Up Study Committee initiated the development of the 
sTurvey instruments (see Appendix A for revised instruments). Staff f roi* each 
of the college's program areas assisted in developing questionnaire items and 
open-ended questions which were relevant to their area of concern. As a 
result of extensive staff participation, content validity can-be claimed for 
the questionnaire items. 

The committee agreed that a common instrument should be developed which 
would allow for inter-program comparisons'. The?, major assumption underlying 
the use of the same instrument to evaluate different program graduates was 
that a given type of work or work situation involves generally the same "job 
requirements" irregardless of the activity or the situation. 

In addition to the questionnaire items, an interview-schedule was de- 
veloped to enrich the data analysis and to gather data specific to individual 
programs. Also, the interview-schedule allowed data gathering flexibility 
for work requirements specific to individual programs which may not have 
been covered in the questionnaire. Using the critical incident technique 
(Flanagan, 1954) , two interview items were developed to ascertain if the 
structured questionnaire items were relevant to all programs*. Other inter- 
view questions pertained to the employee's chances for advancement and the 
employee's overall strengths and weaknesses* 

The population of employers, who hired graduates of occupational pro- 
grams between 1969-71, involved in the study had been identified by career 
graduates from those years in a follow-up survey conducted during fall, 1972. 
Names of employers for graduates not identified in the survey were provided 
by program coordinators,. 



All employers were sent a copy of the questionnaire. Over 80 per cent 
of the employers returned questionnaires. In addition to completing the 
questionnaire, a sample of these employers was randomly selected for the 
interview phase of data gathering. Employers who were to be interviewed 
wore notified that they would be telephoned for an appointment by an MVCC 
staff member for the interview phase of the data gathering. Interview- 
schedules were completed for 60 per cent of the sample. 
R qgutts and Program Recommendations 

Staff at MVCC have conducted two employer follow-ups using the 
specified procedures. The following results are based on the first survey 
conducted spring, 19721 Data from spring, 1973 survey are currently being 
analyzed. * 

Examples of data display are presented in Figures 2 and 3. Figure 2 
displays a composite mean score value for the program areas and the re- 
lated skill areas. Small differences existed between the three program 
areas. Figure 3 displays an example of an individual skill area (technical 
skill) and the rank order of the individual* program areas. 

Results for each of the occupational programs (e.g., business-mid- 
management) were tallied and communicated to the program director, coordina- 
tor and staff involved in that program. No attempt was made to make any 
further statistical analysis on individual programs because of the small 
number of graduates evaluated for each program. A case study describing 
how data on an individual program resulted in course revision can be found 
in Appendix B. 

Data gathered from the interview-schedule were limited because inter- 
views were conducted in only the business and public service areas. How- 
ever, information gathered in the interview procedure did lead to some of 
the conclusions included in the final report (see Appendix C for a summary 
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Example of Inter-Program Comparisons 
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of the results). Other data gathered from the interview process were incor- 
Porated 'Into the data analysis summary sections for the questionnaire items. 
On the basis of t^he data collected from the interviews, the questionnaire 
items appeared to be relevant to each of the program areas and represented 
the range of skills the employer desired. 

In order to examine the nature of the major skill areas, a factor 
analysis of the data was performed. Because complete data were not avail- 
able for all subjects, Veldman's (1967) missing data option was used with 
his principal component-varimax rotation program (eigenvalue set at 1.00). 

Factor analysis of the original 25 item questionnaire designed to 
evaluate technical, human relations and problem-solving skills revealed 
that employers evaluated graduates using a two-dimensional structure which 
included only technical and human relation skills. Factors and their cor- 
responding loadings for each item ate displayed in Appendix D. A total 
of 56 per cent of the variance was accounted for by these two factors. 

The Follow-Up Committee made several changes in the original ques- 
tionnaire for the second follow-up in spring, 1973. Changes in the ques- 
tionnaire we--: the tl creativeness ,t item in the technical skill section 
was. rewritten; the human relations items were split into two sections 
one relating to communication, the other relating to interpersonal rela- 
tions. No changes were made in the "problem-solving section. 11 

Another factor analysis of the revised instrument has been performed 
and the results from the 1973 survey confirm the previous finding of a 
two factor structure of human relations and technical skills (see 
Appendix E for the item loadings). 

As a result of the entire data analysis after the first follow-up 
several program recommendations were formulated by the Follow-Up Co mmi ttee. 



The recommendations were: 

A. For all occupational programs, review the written communica- 
tion objectives for Com 111 and 112, and develop recommenda- 
tions for implementing changes in these objectives or student 
learning activities. 

B. Human relations objectives should be reviewed and recommenda- 
tions developed for implementing changes in these objectives 
for all career programs. 

C. For health science programs, review the program objectives 
related to equipment "creativeness," and develop recommenda- 
tions for implementing changes. 

D. As subsequent data warrants, the general career program ob- 
jective of developing entry level skills only should be 

evaluated in light of possible objectives for career advance- 
rs - 

raent. vJ 
Problems and Revisions 

The following problems have been experienced in conducting the employe 
follow-up. 

~ • The interview phase of the employer follow-up is a difficult 
one for the program coordinators and directors to carry out. 
Because of the time consuming nature of the interviews as 
well as the difficulty in making appointments with various 
supervisors of the employees, the interview phase hao resulted 
in limited data collection. 
• Employers 1 responses to the questionnaire items could be 
characterized as reflecting a "halo effect." This makes in- 
terpretation of the data difficult and conclusions based cn 
the data interpretation are tenuous. 
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• In the data analysis ph&je, the lack of an agreed upon 
criterion of what is ,f excu*xent ,f and what is "good" has 
made data interpretation difficult. 

• The instruments* reliability and predictive validity need 
to be established. 

In an effort to further develop the employer evaluation system, the 
following revisions are being contemplated: 

• Explore other instrument forms* For example, randomly 
select items which could be used for inter-program com- 
parison and develop specific itv.ms for the individual 
career programs; develop graphic rating scales; tailor 
the items for each program area and compare on the skill 
area concept. 

• For each program, establish performance standards against 
which results can be compared. 

Positive Outcomes 

The employer follow-up has stimulated a number of positive outcomes: 

• High ratings for the training of occupational graduates 
provide empirical evidence that MVCC occupational pro- 
grams are meeting the needs of the employers. 

• Interviews conducted by MVCC staff have resulted in 
better relations between staff and community employers. 
Future intership contacts and employer input into MVCC 
career program have resulted. 

• For specific occupational programs, recommendations for 
review of specific course objectives have been made. 



In spite of a few time consuming problems, the benefits far outweigh 
the problemr developing an employer evaluation system. As a result of 
this survey. Moraine Valley has greatly enlarged its evaluative data base 
for its career programs* 



// 
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Appendix A 

MORAINE VALLEY COMMUNITY COLLEGE 
Employer Follow-Up 

Provide the following information about the employee listed^below. 

Name of Employee: - Date: 

Employer: ■ " ; " 

Job Title of Employee: ; ' 

Description of Duties: : 

Title of Person Evaluating Employee: - 

******************* ** ********** ********* 

A. 1. Is the above named employee still in your employ? Yes No 

2. If not, was his (her) termination voluntary? Yes ' No " 

3* Reason for termination: 

4. Approximately how long is (was) the person in your employ? 

**************************************** 

R. Technical Skills 

For items 1-5, rate the following technical skills by circling the letter 
before the number of each statement. Use this key: 

E « Excellent 

G = Good 

F « Fair 

P « Poor 
No = Not Observed 
Na = Not Applicable 



R 


G 


F 


P 


No 


Na 


1. 


Handles equipment or instruments with speed. 


E 


G 


F 


P 


No 


Na 


2. 


Manipulates equipment or instruments with accuracy. 


E 


G 


F 


P 


No 


Na 


3. 


Cares for equipment or instruments. 


E 


G 


F 


P 


No 


Na 


4. 


Adapts equipment for new tasks. 


E 


G 


F 


P 


No 


Na 


5. 


Aware of equipment's capabilities. 
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For the remaining items, rate the following skills by circling the letter 
before the number of each statement. Use this key: 

He - Highly Effective 

E ■ Effective 

I » Ineffective 
Hi - Highly Ineffective 
No ■ Not Observed 
Na = Not Applicable 

C. Human Relations 

He E I Hi Na* 6. Cooperates with fellow workers to get job done. 

He E I Hi No Na 7. Cooperates with supervisor and other higher 

officials. 

He E I Hi No Na 8. Presents ideas and recommendations to persons 

or groups in a non-offending way. 

He E 1 Hi No Na 9. Promotes the use of new ways of doing things. 

He E I Hi No Na 10. Develops an acceptable course of action when 

different points of view are presented. 

He E I Hi No Na 11. Accepts criticism without becoming resentful. 
***************** * * ********************* 

D. Communication Skills 



















He 


E 


I 


Hi 


No 


Na 


12. 


Organizes thoughts in writing. 


He 


E 


I 


Hi 


No 


Na 


13. 


Uses appropriate grammar and spelling i 
writing. 


He 


-E 


I 


Hi 


No 


Na 


14. 


Adapts writing to the audience. 


He 


E 


I 


Hi 


No 


Na 


15. 


Poised when speaking to groups. 


He 


E 


I 


Hi 


No 


Na 


16. 


Accomplishes tasks in group situations. 


He 


E 


I 


Hi 


No 


Na 


17. 


Listens to viewpoints of others. 


He 


E 


I 


Hi 


No 


Na 


18. 


Asks questions which clarify task. 


He 


E 


I 


Hi 


No 


Na 


19. 


Helps people who ask for assistance. 


He 


E 


I 


Hi 


No 


Na 


20. 


Makes himself accessible to others. 
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He - Highly Effective 

E = Effective 

I = Ineffective 
Hi - Highly Ineffective 
No - Not Observed 
Na = Not Applicable 

E. Problem, Solving Skills 



He 


E 


I 


Hi 


No 


Na 


21. 


Ability 


to 


recognize a problem. 


He 


E 


I 


Hi 


No 


Na 


22. 


Ability 


to 


define a problem. 


He 


E 


I 


Hi 


No 


Na 


23. 


Ability 


to 


consider alternative solutions.' 


He 


E 


I 


Hi 


No 


Na 


24. 


Ability 


to 


implement a successful 'solution. 


He 


E 


I 




No 


Na 


25. 


Assigns 


time for carrying out the various . 



work activities (scheduling). 

He E I Hi No Na 26. Combines others' efforts into a common 

action (coordinating) . 

He E I Hi No Na 27. Divides work into individual jobs and provides 

a method of blending the individual efforts 
(organizing). 

He E I Hi No Na 28. Uses present and past information to develop 

a future course of action (planning). 

************************** ************** 
F. Additional Comments: 
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Appendix A 

MORAINE VALLEY COMMUNITY COLLEGE 
Employer Interview Schedule 



Name of Employee: Date: 

Employer : 



Name of Person Evaluating this Employee: 

Title: 



********************* *** * * * ************** 

1. "Describe a situation when tKis employee performed a job-related task very 
effectively / 1 

"Next,* specify the abilities, techniques and skills this employee used in 
performing this task." 

(Record these* responses in verbatim. ) * * 



2. "Describe a situation when this employee performed a job-related task very 
ineffectively » 11 

"Next, specify the abilities, techniques and skills this employee should 
have used in performing this task." 

(Record these responses in verbatim*) 
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3. "What has been (was) this employee's greatest strength? 11 



4. "What has been (was) this employee's greatest weakness?! 1 



5. "Please tell me the number of the answer choice which best states your 
evaluation." (H<md employer answer card, side 1 up.) 

(Say) "This. employee's chances for advancement, compared with your other 
employees in similar positions, are (were)?" (Circle employer's response.) 

1 - Better 2 - Same 3 - Not as Good 4 - *No Chance at All' 5 - No Opinion 

6. "Use the other side of the response card (side 2). Based on your observa- 
tions of this employee's training, would you employ more MVCC~ graduates for 
this same position?" 

1 - Definitely 2 - Maybe 3 - Definitely Not 4 - No Opinion 

7. "Other comments on this graduate's preparation for this position:" 



8. "Does the employer desire a copy of this final report?" 

Yes 

No 

Concluding Remarks 

"Your responses in no way individually reflect upon your employee. We are under- 
taking a thorough evaluation of our programs and will strengthen them whenever 
the results of interviews such as this one indicate the need." 



Pick up the questionnaire. If the questionnaire is not complete, ascertain when 
it will be finished. Leave a Moraine Valley return envelope if questionnaire is 
not completed. 



Name of Interviewer: 
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Appendix B 

4 

Case Study of Shorthand Course Revision 

This case study describes how data gathered from the interview-schedule of 
the employer follow-up survey was used in program revision. / 

Staff from the business program area, which includes secretarial science 
courses, conducted a series qf interviews with supervisors of business 
program graduates during the spring of 1972.* Among those interviewed 
were employers of MVCC secretarial science graduates. Using the inter- 
view-schedule, employers were asked to respond to strengths and weaknesses 
as well as to describe activities which they felt the student had performed 
adequately and inadequately. Analysis of responses to these general* open- 
ended questions revealed two deficiencies in the shorthand training of our 
secretarial science graduates: proficiency level (speed) and ^the end pro- 
duct (mailable letter). Employers revealed that the students' speed at 
4 taking shorthand as well as being .able to produce a mailable letter di- 

rectly from the dictation were not being accomplished. 

i The director of the business-related programs, after reviewing the data; 

called a meeting with the secretarial science program coordinator and the 
instructors. At this meeting, the instructors agreed that their students 
were not getting enough timeTin skill building activities. The instructors 
recommended to the director that an additional laboratory hour per week be 
, established to allow the student to develop wore adequate skills in taking 
shorthand. 

To determine the amount of laboratory time other area community colleges • 
were devoting to shorthand, the director -of business-related programs con- 
ducted a survey. Based on data from this survey, the director was able to 
determine that Moraine Valley offered fewer lab contact hours for secre- 
tarial science students than other area colleges. 

Based on the recommendations of the instructors and on data from the survey, 
a revision in the time allocated to lecture and lab was made. One hour of 
J lecture was deleted and one additional lab hour per week was added to the 

secretarial science shorthand course to overcome the two deficiencies. 

J In conclusion, information which was gathered through an informal interview 

revealed program weaknesses Which heretofore had not been articulated. The 
structured questionnaire had not revealed these weaknesses either. Only 
through the interview situation were the weaknesses revealed in the training 

f of MVCC secretarial science graduates. 
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Appendix C 

Conclusions from 1972 Employer Follow-Up* 



A. Employers rated as "effective 11 or "highly effective" the training in 
technical skills, human relations and problem-solving that their 
employees received at Moraine Valley Community College. 

B. Evaluations of the training for public service, health and business 
program areas were similar. 

C. In. the evaluation of technical skills, public service employees were 
rated slightly higher than graduates of business and health programs. 

D. "Cooperation with fellow workers" by public service graduates received 
the highest rating of the human relations skills. Written communica- 
tion by health graduates was rated the lowest. 

E. "Coordination" (problem-solving skill) was rated higher for business 
graduates than for graduates of other programs. 

F. Personality factors were mentioned most often by the supervisors as 
areas of strength or weakness. 

6. The chances for advancement of half the MVCC program graduates were 
rated higher than the chances of other employees in similar positions. 

H. Supervisors indicated an interest in hiring MVCC graduates. 

I. A positiv^ correlation exists between MVCC evaluation and job perfor- 
mance evaluation. 



Taken from: Employer Evaluation of Occupation Graduates , Moraine Valley 
Community College, Palos Hills * Illinois, 1972. 
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Appendix D 



Rotated Factor Loadings 
of Employer Follow-Up for Spring 1972 



(Human Relations) 



Alternative solutions 
New ways of doing things 
Problem recognition 
Planning 
Listens 

Acceptable course of action 

Implement solution 

Organizing 

Accessible 

Oral presentation 

Scheduling 

Helps people 

Problem definition 

Coopera ti on — Wo rkers 

Asks questions 

Writing 

Non-offending ideas 
Accepts criticism 
Coordinating 



(Factors) 



II 



.726 
.704 
.701 
.694 
.686 
.678 
.676 
.674 
.671 
.653 
.618 
.604 
.594 
.587 
.586 
.562 
.495 
.450 
.406 



-.503 
-.321 
-.434 
-.473 
-.269 
-.250 
-.459 
-.443 
-.446 
-.278 
-.376 
-.343 
-.403 
-.392 
-.447 
-.119 
-.359 
-.246 
-.223 



(Technical Skills) 

Handles equipment with speed .232 -.876 

Equipment maintenance .287 -.767 

Manipulates equipment with accuracy .538 -.671 

Cooperation — Supervisors .331 -.655 

Equipment orientation .399 -.652 

Creative use of equipment .495 -.619 



Per cent of variance accounted for was: 56.09% 
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Appendix E 

Rotated Factor Loadings 
of Employer Follow-Up for Spring 1973 



I II 

(Human Relations) (Factors) 

Organizing ' . 855 - . 520 

Coordinating .791 -.462 

Consider alternative solutions .776 -.498 

Scheduling .735 -.513 

Problem definition .735 -.435 

Promotes new ways -659 -.372 

Organizes thoughts (writing) .637 -.188 

Problem recognition .633 -.415 

Accomplishes — group situations .626 -.322 

Grammar, spelling .612 -.424 

Acceptable course .605 -.397 

Cooperates — fellow workers .603 -.095 

Asks questions .592 -.427 

Implement solutions .591 -.418 

Non-offending ideas .589 -.220 

Adapts writing (audience) .581 -.368 

Holps people .579 -.348 

Poised— speaking .552 -.317 

Accessible to others .534 -.304 

Cooperates — supervisor .518 -.010 

Planning .497 -.299 

Accepts criticism .487 -.324 

Listens to others .471 -.237 



(Technical Skills) 

Adapts equipment for new tasks 
Handles equipment with speed 
Aware of equipment capabilities 
Manipulates equipment with accuracy 
Cares for equipment 



.329 
.264 
.354 
.317 
.313 



UNIVERSITY OF CALIF. 
LOS A f ulES 

JUL 2 7 1973 

CLEARINGHOUSE FOR 
JUNIOR COLLEGE 
INFORMATION 

-.927 
-.884 
-.742 
-.735 
-.686 



Per cent of variance accounted for was: 57% 



